Towards an assistive peripheral visual prosthesis for long-term treatment of retinitis pigmentosa: evaluating mobility performance in immersive simulations

2.1.1. Software components and image processing

To design a virtual outdoor obstacle course, a freely available computer game development engine (CryEngine 3.5.6, Crytek, Frankfurt, Germany) was used to create photorealistic environments, employing state-of-the-art, credible physics and artificial intelligence for enhancing meaningfulness of SPV research. CryEngine's open-source C++ code was modified and extended to integrate tracking of angular head motion from a head-mounted display (HMD). Thus, the participants' head scanning movements could be incorporated into the virtual scene gaze control (see Hardware specifications). The participants' position, walking speed, head orientation and motion, distance travelled, and collisions within the environment were recorded along with the corresponding timing of each these parameters.

Customized software for SPV and SRV written by the authors (MPZ, PM) in C#/C++ allowed for real-time capture of a video stream, image processing and subsequent presentation as phosphenized vision on a display. For this study, image workflow was as follows: the CryEngine environment was displayed on a computer screen with 1920 × 1080 pixel resolution at 75 Hz. The FOV of the scene was set to 160° × 90°. The screen content was continuously captured via a webcam at 640 × 400 pixel resolution and 30 frames per second (fps). Due to the difference in aspect ratios (16:9 versus 16:10), the webcam was positioned centrally to exactly capture the vertical extent of the screen, while discarding the far left and right portions, thus capturing a scene angle of 144° × 90°. The webcam stream was then processed by the SPV/SRV software (see 'SPV and SRV parameters' section) at 30 fps. Using the freeware Deskope [39], the processed image stream subsequently underwent up-sampling to 1280 × 800 pixel resolution and a series of image transformations (split-screen stereo rendering, distortion correction) to render to the stereoscopic display of the HMD at 60 Hz. Please see [40] for details. Since the HMD has a FOV of approximately 117.4° × 112.5°, as part of the rendering process, the SPV/SRV image was scaled to and centred within this FOV in order to ensure correct scene representation, and that the FOV of the scene part visible on the display matched its intrinsic FOV.

The webcam, being the component with the lowest resolution and sampling rate, determined the functional resolution of the final image on the HMD. Image capture and display at native HMD resolution would have been possible in theory; yet this would have introduced an extensive time delay within the series of already computationally demanding image transformations. The image processing and interaction between CryEngine and the HMD attributed to latencies less than 100 ms; a potential impact on task performance would have likely equally affected all experimental conditions. Figure 1 illustrates the hardware and software setup as well as the flow of data for image and motion processing.

Zoom In Zoom Out Reset image size

2.1.2. Hardware specifications

2.1.3. SPV and SRV parameters

Within the SPV/SRV software, multiple parameters such as phosphene size, separation and position could be freely adjusted by the experimenter as required for the combinations used in the study. Additionally, residual vision was displayed as circular tunnel vision of desired extent, adjusted to simulate the extent of retinal degeneration. Illustrative examples of the combined residual/phosphene images presented to the participants are shown in figure 2. For simulation of a fixed, central residual VF of 10° diameter, a black mask was applied to the displayed image stream, only omitting the circular central 10° portion. Along the boundary between SRV and mask, within the zone of 4–5° eccentricity, a transition zone of decreasing visual acuity (VA) due to progressing, partial photoreceptor loss was simulated by linearly increasing mask opacity from zero to maximum. Peripheral prosthetic vision was displayed in a hexagonally arranged mosaic of 100 simulated phosphenes in the inferior VF surrounding the SRV. An aim was to investigate sensible distributions of a limited electrode count similar to that of the central retinal prosthesis prototype of the authors, which reflects the current state of technology in implants which receive stimulation instructions from external vision processing units [41]. Three different array configurations were used in this study (figure 2):

• (1)  
  'Higher-acuity layout'. Phosphene separation: 3°; minimum eccentricity: 10.6°, maximum eccentricity: 25.6°; horizontal/vertical extent: 51°/20.8° (figure 2(b))
• (2)  
  'Wider-angle layout'. Phosphene separation: 6°; minimum eccentricity: 21°, maximum eccentricity: 51.2°; horizontal/vertical extent: 102°/41.6° (figure 2(c))
• (3)  
  'Eccentricity-dependent layout'. Phosphene separation: increases with eccentricity³ . Minimum eccentricity: 13° maximum eccentricity: 48.2°; horizontal/vertical extent: 86°/38° (figure 2(d))

Zoom In Zoom Out Reset image size

This SPV/SRV model was to represent the perception experienced shortly after implantation of an electrode array, placed at a distance to functional photoreceptor tissue to not compromise residual vision. Therefore, a gap between SRV and SPV was introduced, its minimum extent (from 5° to 10.6° eccentricity in the higher-acuity layout) chosen to approximately equal the SRV radius itself.

Brightness of single phosphenes was determined by image luminance around their respective locations, using Gaussian-weighted filtering, which has demonstrated advantages over point sampling and regional averaging in visual fixation and smooth pursuit [43]. Filter width (σ value) was set to 66% of the inter-phosphene separation angle. This amount of overlap of Gaussian kernels has been shown to maximize information content of the prosthetic image [44].

Image clarity and the ability to discern between scene features are mainly determined by the range of luminance values present in the (phosphene) image, i.e. by scene contrast. Real-time 'contrast stretching' was applied to take advantage of the full dynamic range of the phosphene vision in low-contrast conditions [45]. Phosphene luminance levels were continuously normalized to cover the whole range of possible values, with the lowest luminance level found matched to a phosphene brightness of 0 (no phosphene, black), and the highest level set to 255 (brightest phosphene, white).

Each simulated phosphene was rendered using a Gaussian intensity profile (figures 2(b)–(d)).

Approval for this study was given under the authority of the UNSW Human Research Ethics Committee. Eleven participants having not previously experienced SPV were recruited (seven female, four male, mean age 25.1 ± 4.2 years) and invited to sit for nine weekly sessions. Session duration was limited to approximately one hour each to ensure participant comfort.

2.2.1. Experimental procedure and vision conditions

2.2.2. Virtual mobility tasks

The course consisted of four distinct tasks (figure 3).

Zoom In Zoom Out Reset image size

Safely navigating around low-lying obstacles is one of the greatest challenges for people with restricted peripheral VF due to RP [47, 48]. In the 'low-lying obstacles' task (figure 3(a)), participants were asked to walk along a sidewalk and to circumvent ten low-lying obstacles (road cones) in an alternating, slalom-like fashion. Emphasis was placed on avoiding collisions and correct alternation of side of crossing, and walking distance and time was to be minimized provided subjects were able to do so safely. Slalom movement required participants to perceive and relate proximal objects to their own position. This is analogous to daily situations where multiple obstacle positions have to be considered in order to estimate a safe walking trajectory. Before each run, road cone position was randomized within ±1.66 m perpendicularly, and ±0.83 m along the sidewalk, the range defined by the extent of the sidewalk.

The 'static pedestrians' task (figure 3(b)) included narrowing corridors and nine static pedestrians. Participants were asked to traverse the corridors while avoiding collisions with the pedestrians. To induce the necessity for continuous re-orientation and re-focusing on obstacles in the restricted environment, participants had to perform U-turns after every three pedestrians to further advance through the course. Before each run, pedestrian position was randomized within ±1 m perpendicularly, and ±5 m along the sidewalk, this range defined by corridor width and inter-pedestrian distance.

People with RP are largely concerned about 'moving about in crowded situations' [48]. In the 'moving pedestrians' task (figure 3(c)), participants were asked to walk a dynamic scene along a straight sidewalk, with 27 pedestrians moving along the length of the sidewalk. Participants were asked to avoid collisions. Movement decisions had to be continuously evaluated based on virtual pedestrian behaviour. Pedestrians were programmed to dynamically stop, evade and then choose a new path as soon as they came closer than 1.5 m to the participant, having a reaction time of 0.5 s. This was to ensure collisions occurred solely due to the participant walking into pedestrians rather than vice versa.

In real street scenes, paths are frequently highlighted by lines of variable visibility, which run not necessarily at right angles. Finding a path between obstacles, such as in a parking lot, is a common task. The 'path following' task (figure 3(d)) involved following a regularly intersected line on the floor as closely as possible and as fast as comfortable between large obstacles (cars). Eight cars were placed laterally to the path and in addition to being obstacles, further served as visual cover, preventing participants from overlooking the more distant line trajectory. On each iteration, one of five line-car arrangements of similar length was randomly chosen.

The four tasks were performed in the sequence a, b, d, c, resembling sub-sections of the continuous obstacle course.

Participants were able to adjust their velocity freely up to a maximum of 2.5 m s⁻¹. This high upper limit was chosen to not restrict participants in case they felt sufficiently comfortable to navigate at higher speeds, yet without making navigation unstable due to abrupt movements.

Throughout the experiment, the following raw data were continuously recorded and written to text files: participant position, speed, and collisions with any part of the environment. Data were imported into Matlab (R2014, Mathworks, Natick, MA, USA) for processing using custom software written by the author (MPZ) to determine motion and incidence variables, representing different objective measures of performance. In particular, these were: percentage preferred walking speed (PPWS), distance walked and time required to complete the tasks; and types of objects involved in collisions. Determining the number of collisions, repeated contacts with the same objects were counted multiple times if they occurred at least 0.5 s apart. In addition, the number of errors in the slalom course, i.e. deviations from an alternating left–right–left/right–left–right cone circumvention pattern, usually by missing or misjudging the position of single cones ('low-lying obstacles' task), was determined. Also, the average deviation in metres from the line ('path following' task) was acquired by continuously recording the Euclidian distances between participant position and the closest line coordinate, for each acquired data point. For the 'path following' task, since possible line trajectories differed in length, distance walked and time taken were normalized to a path length of 100 m.

PPWS has been introduced by Clark-Carter et al [46] and widely used to compare the effect of different aids in improving mobility performance. The PWS is the optimal speed that people with visual impairment would walk if their vision were fully restored. PWS has been originally determined for visually impaired people by letting them walk with the help of a sighted guide [46] or walking a straight unobstructed route [49, 50]. In the case of the present study the method of calculating this measure required adaptation as the participants all had normal vision so they were instructed to walk the same route with a full view of the visual environment on the HMD, i.e. one that was not restricted in VF by SPV or SRV, and their preferred speed assessed. The ratio of speed with the full view to the speed with the SRV or residual vision with assistive phosphene arrays was calculated. As compared to absolute walking speeds, PPWS allows participants to act as their own controls, normalizing data for differences in age, height, and physical fitness. In this study PWS was determined before each session, so normalizing for between-session variations in performance was established (see supplementary table 1 for observed PWS).

Processed data were then exported to SPSS 22 (IBM, Armonk, NY, USA) for statistical analysis. Approximate normality of the data subsets was assessed by visual inspection of Normal Q–Q plots [51] and histograms. Homogeneity of variance was ensured using Brown–Forsythe tests [52]. In case data followed an approximately normal distribution, parametric testing was chosen. Prior to investigating the data, we hypothesized that using assistive phosphene lattices would provide a significant difference in the means of the investigated variables as compared to plain residual vision, and that some layouts might be significantly better than others. A statistical test for pairwise comparisons was required. Due to the novelty of combining SPV and SRV and the variety of tasks and variables tested, potentially detrimental effects of assistive SPV as compared to SRV alone could not be ruled out. A two-sided test sensitive for effects in both directions was therefore chosen. Tukey honestly significant difference (HSD) tests with correction for multiple comparisons were performed [53]. SPSS required a preceding analysis of variance; here, a mixed model was chosen [54], factoring in the effects of vision condition and repetition (fixed factors) as well as controlling for subject (random factor). The Tukey HSD test is justified to be 'stand-alone' since it already protects against Type I errors [55–59]. A significance level of p < 0.05 was used. In case data deviated from a normal distribution, which was the case for the measures of task time, distance walked and deviation from line, the aligned rank transformation [60] was applied prior to Tukey tests. This method allows for a non-parametric analysis by applying ranks over the data set while a subsequent data alignment process simultaneously controls Type I error rate for multifactorial models. In the following section, graphs depict observed, non-ranked data (mean ± standard error) and statistical test results (asterisks, brackets) for each of the presented vision conditions. For PPWS, the values for 'fv' are not shown since they are implicit in the measure. In the supplementary tables, p values derive from the respective parametric or nonparametric test procedure applied.

Participants made significantly less normalized errors using assistive phosphene with residual vision as compared to plain residual vision in the 'low-lying obstacles' task (figure 4(a)). This was the case for all three of the phosphene layouts in combination with 10° of residual vision. Errors counts were lowest using the wider-angle layout, followed by the higher-acuity layout and the eccentricity-dependent layout, as compared to plain residual vision.

Zoom In Zoom Out Reset image size

Mean PPWS, as compared to full vision was found to be significantly higher for all vision conditions that utilized phosphene lattices (figure 4(b)) than when plain residual vision was presented. The highest gains were found using the higher-acuity layout, followed by the eccentricity-dependent and the wider-angle layout. Also, PPWS using the higher-acuity layout was significantly higher than with the wider-angle layout.

The mean times needed to complete the task (figure 4(c)) were significantly reduced from residual vision only using the eccentricity-dependent layout, even more so with the wider-angle layout, and the fastest mean time was achieved with the higher-acuity array. The latter layout also afforded the participants with significantly improved performance over the other two layouts.

Notably, only when using the higher-acuity layout did participants walk significantly shorter distances than with plain residual vision (figure 4(d)). Paths were also significantly shorter than when using the other two layouts.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Contrasting the faster pace and shorter trajectories when using the higher-acuity layout, participants experienced a significantly lower average number of collisions with the ten low-lying obstacles only when using the wider-angle and eccentricity-dependent layouts, as opposed to both plain residual vision and the higher-acuity layout. See figure 4(e). Supplementary tables 2 and 3 show statistical test results for all variables measured in this task as well as observed mean values and standard errors.

Walking speed, completion time, distance walked, and collisions in the 'static pedestrians' task are shown in figures 5(a)–(d). Average task time using the higher-acuity layout was significantly reduced as compared to plain residual vision (figure 5(b)). Participants also incurred significantly fewer collisions with static pedestrians using the higher-acuity layout (figure 5(d)), than with plain residual vision. See supplementary tables 4 and 5 for statistical test information as well as observed means and standard errors for all variables recorded for the 'static pedestrians' task.

Walking speed, completion time, distance walked, and collisions in the 'moving pedestrians' task are shown in figures 6(a)–(d). Moving along a sidewalk with 27 moving pedestrians, reductions in PPWS using peripheral phosphene lattices as compared to plain residual vision (figure 6(a)) were statistically significant for the wider-angle layout. In supplementary tables 6 and 7, outcomes for all statistical tests as well as observed means and standard errors for the 'moving pedestrians' task are listed.

Deviation from the line, walking speed, completion time, distance walked, and collisions in the 'moving pedestrians' task are shown in figures 7(a)–(e). Participants were able to significantly increase PPWS with peripheral phosphenes using the higher-acuity and wider-angle layouts (figure 7(b)). With all phosphene layouts augmenting residual vision, significantly less time was needed to reach the end of the path (figure 7(c)). Participants were able to significantly reduce collisions with the cars along the path using higher-acuity and eccentricity-dependent layouts (figure 7(e)). See supplementary tables 8 and 9 for statistical test outcomes as well as observed means and standard errors for all variables recorded in the 'path following' task.

The benefits of the proposed assistive prosthesis layouts were found to be task-dependent, which also applies to many other mobility aids.

4.1.1. Low-lying obstacles

4.1.2. Static pedestrians

4.1.3. Moving pedestrians

4.1.4. Path following

Interestingly, the lower the PPWS when using residual vision only, the more pronounced the beneficial effects of assistive phosphenes. In the 'low-lying obstacles' (64% PPWS) and 'path following' tasks (55% PPWS) performance was improved by several phosphene layouts across multiple measures. In the 'static pedestrians' task (71% PPWS) performance was only improved by the higher-acuity grid, and not at all for 'moving pedestrians' (89% PPWS). 'Baseline' PPWS values falling below a certain threshold (here, around 70%) might identify candidate tasks for assistive prosthetic vision.

On each start of the obstacle course—within the 'low-lying obstacles' task—there might have been an adaptation period of decreased performance. In case adaptation to the visually most restricted SRV-only condition had been particularly pronounced, its performance detriment could have been overrated. This might have exaggerated benefits of assistive phosphenes for this task. Randomization of task order will be introduced in future studies to counteract such effects.

After identification of candidate scenarios for a peripheral prosthesis, a trade-off between electrode array resolution and the covered VF angle will have to be established. Providing equally higher-acuity vision across a larger VF extent by increasing electrode count is currently not possible due to technical and physiological constraints [65].

Higher phosphene resolutions benefitted navigation under SPV [38, 62, 66]. Likewise, VA contributed to walking and mobility under RP. However, these reports are in line with other researchers stating that VF extent more strongly predicts mobility performance than VA [24, 25, 34, 49, 67–69]. FOV up to 70° [32] can be beneficial for mobility performance. Previous consideration of a wide-field prosthesis [70] reflects this. Studies particularly emphasize the importance of certain VF areas such as the central 37° and the inferior areas for mobility and fall prevention [34, 61, 71, 72]. In self-reported surveys, the central 5° as well as the central 10–30° best predicted self-reported vision-related activity limitations in mobility [73].

Targeting low-contrast scenarios, covering these crucial VF areas with a certain level of VA might produce highest benefit, as reflected by the higher-acuity layout. At higher contrast, maximizing VF does not necessarily improve all aspects of mobility: when circumventing low-lying obstacles, collisions were reduced at the expense of speed. Rather, different goals in accomplishing the respective task are balanced based on the provided visual information.

Previous research has shown the role of optic flow—based movement strategies for walking and in peripheral VF loss [27, 74]. In the higher-acuity phosphene layout, the gap between residual and phosphene vision was smaller, improving optic flow between the two modalities. In the wider-angle and eccentricity-dependent layouts, optic flow was instead available more peripherally, permitting a continuous perception of obstacles during crossing. Potentially, optic flow in the central FOV plays a role for choosing movement speed, while peripheral optic flow is important for judging safe distances when passing by objects. This might explain the lower low-lying obstacle collision count with more eccentric phosphenes. Ultimately, despite the aforementioned benefits of higher-acuity layouts, prevention of incidences might be more important than task completion speed, implying that visual angle should be kept at a maximum.

Foreseeing this conflict, the hybrid, eccentricity-dependent layout featured a smooth transition of resolution. Yet, results did not always favour this geometry. We hypothesized that, due to its higher complexity, learning might have not been completed during the course of the study. Regression analysis however disproved this assumption: on average across subjects, from within the second half of the study (repetitions 9–16), slopes of best-fit lines were not significantly different from zero across all variables, conditions and tasks. Alternatively, the distribution of phosphenes in the VF might have been non-optimal. Potentially, recipients rather favour a clear split between a number of central, high-acuity phosphenes for detecting finer details, supported by a range of high-eccentricity, low-acuity phosphenes catching movement or contrast differences in the periphery.

The phosphenes displayed in simulations like the one presented are idealized representations of the percepts feasible in an actual implant. Guidelines for adequate simulation in order to address this discrepancy have been proposed previously [75], to which we adhered. Even idealized representations can have close correlation to clinical findings [16]. Nevertheless, the achievable clinical benefit of peripheral electrode arrays might be less pronounced than reported here. For instance, with ten discernable phosphene luminance levels reported clinically [9], the number of levels in this simulation represents a best-case scenario.

Similarly, virtual and real-world navigation strategies might differ. In the latter, collisions potentially have serious consequences; collision avoidance is likely favoured over speed. Follow-up studies should centre on the how deviations from ideal phosphene and real-world scene perception influence the achievable benefit.

One limitation was that no monitoring was performed as to whether participants complied to fix their gaze at the centre of the display at all times. Eye tracking for gaze-locked viewing has been used previously in SPV [76–78]. However, attaching an eye-tracking device in a suitable angle between the tightly fitted HMD and the participant's eyes was infeasible. Further, the more extensive phosphene layouts covered a major portion of the available FOV. Constantly repositioning the SPV and SRV on the display at the centre of gaze direction, larger gaze angles would have resulted in cutting off phosphene fields of these layouts at the edge of the display. Similarly, restricting user eye gaze within certain angles to prevent this would not have benefitted realism. Lastly, at notable display latencies, such as present in our setup, large saccades are reportedly difficult to track [77].

The SRV, the highest-resolution part of the VF, would have been predominantly directed to the most salient scene features, and thus largely hold participant attention and fixation, as these are linked [79]. Within these 10° of SRV, it is imaginable that participants applied saccades to centre gaze on salient features. This might have increased performance similarly across all presented conditions. In case saccades elevated SRV-only performance more strongly and thus diminished the difference to the SRV + SPV conditions, the benefits of peripheral SPV reported here might have been more pronounced with gaze locking applied.

Wandering of the gaze to the SPV might as well have occurred, possibly lowering performance with assistive phosphenes, due to the predominantly lower visual information obtainable relative to the residual vision.

We chose to retrospectively characterize the role of eye tracking. Though software-based, quantitative eye tracking was not feasible, lateral recording of eye movements for qualitative analysis was possible and thus performed in a separate (unpublished) study—on three independent participants while being presented the wider-angle and higher-acuity layouts. Instead of displaying SRV, participants were asked to fixate a white pixel in the centre of the display while performing the 'low-lying obstacles' task. Even without the high-acuity central stimulus, we found that subjects typically fixed their gaze reliably and did not deviate in a coordinated fashion. Nevertheless, future studies will implement eye tracking, which will be less likely to underestimate benefit provided by assistive phosphene vision.

After implantation, the residual VF will continue to constrict; therefore gap size will further increase, which will also impact on optic flow. Eventually, residual vision will disappear. Since central VF is important, keeping the initial gap between residual and prosthetic vision as small as possible might be advisable. Ultimately, a safe implantation distance of an electrode array not compromising viable photoreceptor tissue will have to be determined. Supra-choroidal prostheses are particularly suitable in this scenario.

This study focused on the evaluation of the benefits of early implantation. Basically, explantation of a peripheral supra-choroidal array in favour of a central array following the disappearance of central residual vision is possible [80]. However, since any surgical procedure carries risks, peripheral designs should maintain a comparable mobility performance to that of a later implanted central array. Given the considerable time span between early implantation and complete central vision loss, it is imaginable that recipients will learn to use peripheral phosphenes to the same effect as central ones. Furthermore, due to the anatomical displacement of target RGCs in the centre of the fovea, elicitation of phosphenes in the very central VF is infeasible. Any implanted 'central' array therefore would likely be offset peripherally to a certain distance, diminishing its advantage in providing 'higher-acuity' central prosthetic vision.

Apart from optimizing electrode array geometry, alternative techniques can improve the perceptual quality of prosthetic vision. Head scanning, in particular increased scanning velocity, can facilitate detecting fine details [81]. Training participants to apply goal-directed head scanning strategies might be particularly useful when the distance between phosphenes in a wide-field prosthesis is relatively high.

Alternatively, implementation of image processing algorithms which detect and emphasize salient features for display is a viable approach to maximize the information content of a low-resolution, wide-spread prosthetic display [76, 82–84]. The authors' external vision processor in development for the current iteration of visual prosthesis is capable of a range of real-time processing steps such as edge detection and depth mapping between image capture and display [85].